Scalable edge compute in a distributed control environment

ABSTRACT

Various systems and methods may be used to implement a software defined industrial system. For example, an edge control node of the industrial system may include a system on a chip including a microcontroller (MCU) to convert IO data. The system on a chip includes a central processing unit (CPU) in an initial inactive state to receive an activation signal from, for example, an orchestration server, and change to an activated state in response to receiving the activation signal.

PRIORITY CLAIM

This application claims the benefit of priority to U.S. ProvisionalPatent Application Ser. Nos. 62/587,227, filed Nov. 16, 2017 and titled“DISTRIBUTED SOFTWARE DEFINED INDUSTRIAL SYSTEMS”, and 62/612,092, filedDec. 29, 2017, and titled “DISTRIBUTED SOFTWARE DEFINED INDUSTRIALSYSTEMS”; the above-identified provisional applications are incorporatedherein by reference in their entirety.

TECHNICAL FIELD

Embodiments described herein generally relate to data processing andcommunications within distributed and interconnected device networks,and in particular, to techniques for defining operations of asoftware-defined industrial system (SDIS) provided from configurableInternet-of-Things devices and device networks.

BACKGROUND

Industrial systems are designed to capture real-world instrumentation(e.g., sensor) data and actuate responses in real time, while operatingreliably and safely. The physical environment for use of such industrialsystems may be harsh, and encounter wide variations in temperature,vibration, and moisture. Small changes to system design may be difficultto implement, as many statically configured I/O and subsystems lack theflexibility to be updated within an industrial system without a fullunit shutdown. Over time, the incremental changes required to properlyoperate an industrial system may become overly complex and result insignificant management complexity. Additionally, many industrial controlsystems encounter costly operational and capital expenses, and manycontrol systems are not architecturally structured to take advantage ofthe latest information technology advancements.

The development of Internet of Things (IoT) technology along withsoftware-defined technologies (such as virtualization) has led totechnical advances in many forms of telecom, enterprise and cloudsystems. Technical advances in real-time virtualization, highavailability, security, software-defined systems, and networking haveprovided improvements in such systems. However, IoT devices may bephysically heterogeneous and their software may also be heterogeneous(or may grow increasingly heterogeneous over time), making such devicescomplex to manage.

Limited approaches have been investigated to utilize IoT devices and IoTframeworks even despite the technical advances that have occurred inindustrial automation and systems. Further, industry has been hesitantto adopt new technologies in industrial systems and automation, becauseof the high cost and unproven reliability of new technology. Thisreluctance means that typically, only incremental changes are attempted;and even then, there are numerous examples of new technology thatunderperformed or took long periods of time to bring online. As aresult, wide-scale deployment of IoT technology and software-definedtechnologies has not been successfully adapted to industrial settings.

BRIEF DESCRIPTION OF THE DRAWINGS

In the drawings, which are not necessarily drawn to scale, like numeralsmay describe similar components in different views. Like numerals havingdifferent letter suffixes may represent different instances of similarcomponents. Some embodiments are illustrated by way of example, and notlimitation, in the figures of the accompanying drawings in which:

FIG. 1A illustrates a configuration of a software defined infrastructure(SDIS) operational architecture, according to a first example;

FIG. 1B illustrates a configuration of an SDIS operational architecture,according to a second example;

FIG. 2A illustrates a configuration of a real-time advanced computingsubsystem deployable within the SDIS operational architecture of FIG.1A, according to an example;

FIG. 2B illustrates a configuration of an edge control node subsystemdeployable within the SDIS operational architecture of FIG. 1A,according to an example;

FIG. 3A illustrates a configuration of a real-time advanced computingsubsystem deployable within the SDIS operational architecture of FIG.1B, according to an example;

FIGS. 3B and 3C illustrates a configuration of cloud computing and edgecomputing subsystems deployable within the SDIS operational architectureof FIG. 1B, according to an example;

FIG. 4 illustrates a configuration of a control messages bus used withinan SDIS operational architecture, according to an example;

FIG. 5A illustrates a first network configuration for deployment of SDISsubsystems, according to an example;

FIG. 5B illustrates a second network configuration for deployment ofSDIS subsystems, according to an example;

FIG. 6 illustrates a dynamically established set of orchestrationoperations in a SDIS operational architecture, according to an example;

FIG. 7 illustrates an industrial control system ring topology diagram;

FIG. 8 illustrates an edge control topology diagram;

FIG. 9 illustrates an edge control node block diagram;

FIG. 10 illustrates an edge control node-based ring topology diagram;

FIG. 11 illustrates data flow through an edge control node-based ringtopology;

FIG. 12 illustrates a flowchart of a method for activating a processorof an edge control node according to an example;

FIG. 13 illustrates a flowchart of a method for activating a CPUaccording to an example;

FIG. 14 illustrates a domain topology for respective internet-of-things

(IoT) networks coupled through links to respective gateways, accordingto an example;

FIG. 15 illustrates a cloud computing network in communication with amesh network of IoT devices operating as a fog device at the edge of thecloud computing network, according to an example;

FIG. 16 illustrates a block diagram of a network illustratingcommunications among a number of IoT devices, according to an example;and

FIG. 17 illustrates a block diagram for an example IoT processing systemarchitecture upon which any one or more of the techniques (e.g.,operations, processes, methods, and methodologies) discussed herein maybe performed.

DETAILED DESCRIPTION

In the following description, methods, configurations, and relatedapparatuses are disclosed for the configuration, operation, andadaptation of software-defined industrial service (SDIS) deployments. Inparticular, the following SDIS deployments include features of modernoperational architecture-based industrial systems, along with derivativearchitectures or solution instances of such deployments. For instance,such architectures and instances may include virtualized control serversystems, which implement features of an edge control device and acontrol messages bus within a control or monitoring system. Sucharchitecture and instances may be further integrated with aspects of IoTnetworks, involving various forms of IoT devices and operations.

The processing techniques and configurations discussed herein include avariety of approaches for managing operations, data, and processingwithin various types of SDIS architectures. An overview of the followingapproaches are provided in the following paragraphs; further referenceto specific implementation examples and use cases is discussed below.

In an example, the systems and methods described herein address theproblem of over or under provisioning the compute capability at the edgeof an industrial control system. Over provisioning the compute resourceswastes money, electrical energy, and thermal energy. Under provisioningthe compute resources sacrifices reliability, and the ability to executethe control strategy. The proposed solution enables the end user withthe performance requirement data to “right” size the amount of computeprovisioned in the control environment. Additionally, the provisionedcompute capability is not static and may be adapted to meet the needs ofthe control system as the requirements change. The techniques discussedherein allow a high performance CPU to be activated, from an initialdormant state, in the Edge Control Nodes by a centralized orchestrationsystem that understands the CPU performance needs of the controlstrategy.

Other examples will be apparent from the following drawings and textdisclosure.

Overview of Industrial Automation Systems

Designing and implementing effective industrial automation systemspresents many technical challenges. Because the lifecycle of anindustrial plant in many cases far exceeds the lifecycle of thetechnology that runs the plant, the administration and maintenance costsof technology are often very difficult to manage. In an example, a SDISdeployment may be adapted for dynamic configuration (andre-configuration) of software and hardware resources in industrialsystems through resource abstraction with the following approaches. Suchresource abstraction provides flexibility for updating the configurationwithout removing the industrial system out of service; such resourceabstraction also provides flexibility for updating the industrial systemwith improved capabilities over time.

Use of open architectures and abstracted links between software andhardware in the presently disclosed SDIS approaches provides these andother technical benefits, while allowing vendors to focus on thecapabilities and implementation of a specific vendor application. Thedisclosed open architectures also promote innovation, reduce the cost ofhardware replacement, and eliminate the risk of hardware obsolescence.The disclosed open architectures enable security to be implemented as anintrinsic part of the SDIS, such as through the use of a hardware rootof trust, signed applications, and comprehensive security management.Such configurations enable a simplified control system with inherentsecurity and the capability to easily integrate capabilities over time.These technical improvements, combined with features of openarchitecture and standards implementations, enable the rapid integrationof industrial control within an SDIS.

Some existing approaches such as the Open Group's Open ProcessAutomation Forum have begun development of a standards-based, open,interoperable process control architecture features for industrialautomation, targeting industries such as Food and Beverage, Mining andMetals, Oil and Gas, Petrochemical, Pharmaceutical, Pulp and Paper, andUtilities. The present configuration and functionality of a SDIS and theaccompanying subsystems and techniques may be integrated with use ofthis standard or similar approaches within industrial automation andsystem deployment efforts. Further, the present configuration andfunctionality of a SDIS and the accompanying subsystems may be utilizedin these or other industries. Accordingly, variations and changes to thefollowing implementations will be evident.

FIG. 1A depicts a first example configuration of an SDIS operationalarchitecture. As shown, a control messages bus 112 is used to connectvarious components of the architecture, with such components includingOperational Tools 120, a Control Server (CS) node 130A, Edge ControlNode (ECN) systems 150, Intelligent I/O Controller systems 165, BasicI/O Controller systems 160, Gateway systems 170, and Control Stations115. Various field devices (151, 161, 166, 171) are connected to therespective systems (150, 160, 165, 170). Some of the example use casesand configurations of this operational architecture are furtherdiscussed below.

In an example, the Operational Tools 120 may include aspects of:procedure development tools, historian tools, human-machine interface(HMI) development, controls, and operations tools. Various aspects ofthe Operational Tools 120 may be implemented with respective virtualmachines 131A operating in the control server node 130A (as furtherdepicted in FIG. 2A).

In an example, the control server node 130A may include aspects ofvarious virtual machines 131A, coordinated via a hypervisor layer 132A,and operating with features of a host operating system 133A and acomputer hardware architecture 134A. The control server node 130A may beused to implement various aspects of orchestration 135A, involving bothmachine orchestration and operational application orchestration. Afurther detailed discussion of the control server node 130A is providedbelow with reference to FIG. 2A below.

In an example, the ECN systems 150 may include various aspects oforchestration (e.g., orchestration implementation) from an ECN I/Ocontroller (e.g., nodes 150A, 150B) operating on specific hardware(e.g., an x86 or ARM hardware implementation). A further detailedexample of the ECN systems 150 and its role in orchestration for variousconnected devices (e.g., field devices 151A, 151B) is provided belowwith reference to FIG. 2B.

In an example, the Intelligent I/O systems 165 may include variousconfigurable aspects of industrial control from an Intelligent I/Ocontroller (e.g., controller 165A, 165B) and an accompanying operatingsystem, used for control or access of various devices (e.g., fielddevices 166A, 166B). Also in an example, the Basic I/O systems 160 mayinclude various operating aspects of industrial control from a Basic I/Ocontroller (e.g., controller 160A, 160B) and an accompanying operatingsystem, used for control or access of various devices (e.g., fielddevices 161A, 161B).

In an example, the Gateway systems 170 may include various configurableaspects for connection to other device networks or deployments, from agateway (e.g., gateways 170A, 170B), used for control or access ofvarious devices (e.g., field devices 171A, 171B). Within the variousdevices, roles of a sensor (“S”) and actuator (“A”) components arelabeled throughout the field devices (e.g., on field devices 151A, 151B,161A, 161B, 166A, 166B, 171A, 171B). It will be understood thatadditional number and types of devices and components may also becoupled to the various systems 150, 160, 165, 170.

The operational architecture depicted in FIG. 1A is configured to enablemany of the same attributes seen in traditional enterprisearchitectures, such as HW/SW modularity, SW portability,interoperability, application extensibility and computationalscalability. Beyond this, the new infrastructure framework componentsintroduced in this architecture, most notably in the implementation ofCS and ECN systems, may be deployed to support both centralized anddecentralized concepts for the SDIS techniques discussed herein.

For example, the use of an ECN I/O Controller (e.g., in ECN nodes 150A,150B) is a significant architecture departure from current DCS(Distributed Control System) and PLC (programmable logic controller)control systems, which have evolved for over the last fifty years. Anyarchitectural advancement in this mission-critical portion of theANSI/ISA-95 automation interface stack must adhere to the strict andresilient requirements of process control. With the SDIS architecturedescribed herein, the ECN system may not only maintain these strictoperational requirements, but also may remain open, interoperable, whileallowing industry uses to safely, reliably, securely and rapidlyintroduce or refresh these systems with ongoing technologicaladvancements. The present SDIS architecture enables wider ecosystemparticipation, innovation and production customization throughout theoperational and control stack. For instance, the ECN system may beprovided with control disaggregation to serve as a basic control systembuilding block, to amplify control function customization and enableincreased process flexibility for a variety of use cases.

FIG. 1B depicts a second example configuration of an SDIS operationalarchitecture. In a similar fashion as shown as FIG. 1A, theconfiguration of FIG. 1B illustrates a control messages bus 112 that isused to connect various components of the operational architecture, withsuch components including cloud components (a real time advancedcomputing system 130B, operating as a control server, and cloudcomputing services 180) edge components (an edge ecosystem 190 withconstituent edge computing nodes 191A, 191B, 191C, a first edgecomputing platform 193, and a second edge computing platform 195), andControl Stations 115. Various field devices (192, 194) with sensors andactuators are connected to the respective edge computing nodes (in theedge ecosystem 190 and edge computing platforms 193, 195). Theoperational goals and features discussed above are also applicable tothe configuration of FIG. 1B.

As a further extension of the SDIS operational architecture introducedin FIG. 1A, the configuration of FIG. 1B illustrates a scenario wherethe operations of the controllers and servers across the various cloudand edge components are virtualized through respective virtual machines,deployed with respective containers, deployed with respectiveapplications, or any combination thereof. As a result, the SDISoperational architecture of FIG. 1B allows a reconfigurable and flexibledeployment to a variety of hardware settings (including both ARM and x86hardware architectures). A further breakout of the real time advancedcomputing system 130B is depicted in FIG. 3A, and further breakout ofthe cloud computing services node 180 and the edge computing node 193 isdiscussed in FIGS. 3B and 3C respectively.

Another aspect of the SDIS architecture may involve the use of real-timecommunications. The control messages bus 112, hosted on a service busfabric 110, may be utilized to enable internetworking convergence onmultiple levels. For instance, the control messages bus 112 may enableuse of Ethernet transports with time-sensitivity, such as throughEthernet-based time-sensitive networking (TSN) open standards (e.g., theIEEE 802.1 TSN Task Group). Further, use of the control messages bus 112may allow greater performance and scale at the cloud server rack leveland across large networked or chassis of edge nodes.

In the SDIS architecture, real-time services may operate on top of areal-time physical transport via the control messages bus 112, such asvia Ethernet TSN. The control messages bus 112 may be adapted to addressthe heterogeneity of existing middleware or communication stacks in anIoT setting (e.g., with use of Open Platform Communications UnifiedArchitecture (OPC-UA), Object Management Group Data Distribution Service(DDS), OpenDXL, Open Connectivity Foundation (OCF), or the likestandards), to enable seamless device-to-device connectivity to addressthe emerging implementations of IoT deployments.

In an example, the orchestration management for a SDIS architecture maybe implemented by a Control Server (CS) design. FIG. 2A illustrates aconfiguration of a control server subsystem (e.g., implementing the CSnode 130A) within an SDIS operational architecture (e.g., theoperational architecture discussed above with reference to FIG. 1A).Specifically, FIG. 2A provides a further illustration of the CS node130A and its component virtual machines 131A, hypervisor 132A, hostoperating system 133A, and hardware architecture 134A; as depicted, theCS node 130A is shown as a single node but may include two or more nodeswith many virtual machines distributed across these nodes.

In an example, the CS node 130A may include orchestration 135A that isfacilitated from machine and operation application orchestration. Themachine orchestration may be defined with use of a machine library 136,such as a database for implementing platform management; the operationapplication orchestration may be defined with use of a control functionlibrary 142 and operational application library 144. For instance,control standards design 141 and integrated (and secure) applicationdevelopment processes 143 may be used to define the libraries 142, 144.

In an example, the CS node 130A is designed to host ISA level L1-L3applications in a virtualized environment. This may be accomplished byrunning virtual machines (VMs) 131A on top of a hypervisor 132A witheach VM encapsulating Future Airborne Capability Environment(FACE)-compliant stacks and applications, or non-FACE applications suchas a human-machine interfaces (HMIs), Historians, Operations Tools, etc.In an example, FACE-compliant VMs may provide an entire FACE stack(operating system, FACE segments, and one or more portable components)that is encapsulated in a VM.

The encapsulation means that each VM may have its own virtual resources(compute, storage, memory, virtual networks, QoS, security policies,etc.) isolated from the host and other VMs by the hypervisor 132A, evenas each VM may be running different operating systems such as Linux,VxWorks, or Windows.

To maximize the benefit of virtualization and robustness, related groupsof portable components may be grouped in a FACE-compliant VM and withthe use of multiple FACE-compliant VMs. Using this approach spreads theworkload across the CS hardware and isolates resources specific to thatgroup of components (such as networks), while still allowing theapplications to communicate with other virtualized and physical devicessuch as ECNs through the network. Distributing the FACE portablecomponents across VMs increases security by isolating unrelatedcomponents from each other, provides robustness to failures, allowsindependent update of functions, and eases integration to allowindividual vendors to provide fully functioning VMs into the system.

In a further example, Layer 2 components may be separated from Layer 3components within separate VMs (or groups of VMs) to provide isolationbetween the layers and allow different network connectivity, securitycontrols, and monitoring to be implemented between the layers. Groupingportable components may also provide benefits to integration, to allowmultiple vendor solutions to be easily combined running multiple virtualmachines and configuring the network between them. Also in a furtherexample, additional operating systems such as Windows, Linux, and otherIntel architecture-compatible operating systems (e.g. VxWorks real-timeoperating system) may each be deployed as virtual machines. Otherconfigurations of the presently disclosed VMs within a CS node 130A mayalso enable other technical benefits.

In an example, a cloud infrastructure platform may be utilized in the CSnode 130A, such as a real-time advanced computing system adapted withuse of open source standards and implementations such as Linux, KVM,OpenStack, and Ceph. For instance, the cloud infrastructure platform maybe adapted to address critical infrastructure requirements such as highavailability of the platform and workloads, continuous 24/7 operation,determinism/latency, high performance, real-time virtualization,scalability, upgradeability, and security. The cloud infrastructureplatform also may be adapted to meet software-defined industrialautomation-specific critical infrastructure requirements.

FIG. 2B illustrates an example configuration of a distributed edgecontrol node (ECN) subsystem within an SDIS operational architecture(e.g., the operational architecture discussed above with reference toFIG. 1A). In an example, the ECN nodes 150A, 150B reside in the ISA-95Level 1/Level 2 and are positioned as a fundamental, basic HW/SWbuilding block.

In an example, the ECN nodes 150A, 150B support a single input or outputto a single field-bus device via a sensor or actuator or smart device(e.g., located externally to an ECN cabinet). The ECN devicearchitecture may be extended through an ECN cabinet or rack system thatextends the openness and flexibility of the distributed control systemaddressing wiring, upgrade, and fault-tolerance limitations withexisting proprietary DCS systems. In an example, the ECN architectureoperates in a standard POSIX OS with a FACE-compliant stack implementedas segments or groups software modules. Various approaches fordeployment of these software modules are referenced in the examplesbelow.

The ECN nodes 150A, 150B may support a variety of software-definedmachines for aspects of orchestration and services (such as theorchestrations depicted below for FIG. 6). In an example, the ECN nodes150A, 150B may integrate with various hardware security features andtrusted execution environment, such as Intel® Software Guard eXtensions(SGX), Dynamic Application Loader (DAL), secure VMM environments, andtrusted computing-standard Trusted Platform Module (TPM). In a furtherexample, secure boot may be enabled with fused and protected keymaterial accessed within protected hardware cryptographic engines, suchas Intel® Converged Security and Manageability Engine (CSME) andPlatform Trust Technology (PTT). Additionally, cryptographic functionsmay be made more secure with special hardware instructions for AESencryption and SHA computations. Other forms of security such as anIntel® Enhanced Privacy ID (EPID) may be being adopted across theindustry as a preferred device identity key, which can be enabledthrough automated device registration (e.g., Intel Secure DeviceOnboarding (SDO)) technology for secure, zero-touch onboarding ofdevices. In further examples, the ECN nodes 150A, 150B and othersubsystems of the SDIS architecture may be interoperable with these orother security approaches.

FIG. 3A illustrates a more detailed configuration of the real-timeadvanced computing system 130B deployable within the SDIS operationalarchitecture of FIG. 1B. Specifically, the configuration of FIG. 3Aillustrates the operation of respective virtual machines 131B which mayinclude different deployment types of virtual machines, containers, andapplications, operating on a hypervisor layer 132B. The hypervisor layer132B may be controlled with use of a host operating system 133B, as theVMs, hypervisor, and operating system execute on the hardwarearchitecture 134B (e.g., a commercial off-the-shelf (COTS) x86architecture). The aspects of real time orchestration 135B may beintegrated into all levels of the computing system operation. Thus, ax86 computing system may be adapted to coordinate any of the cloud- orserver-based SDIS functions or operations discussed herein. Otheraspects of functionality or hardware configuration discussed for the CSnode 130A may also be applicable to the computing system 130B.

FIGS. 3B and 3C illustrates a more detailed configuration of cloudcomputing 180 and edge computing 193 subsystems, respectively,deployable within the SDIS operational architecture of FIG. 1B. In asimilar fashion as depicted in FIG. 3A, a series of virtual machines181, 196, hypervisor layers 182, 197, host operating systems 183, 198,and COTS x86 hardware architectures 184, 199 depicted in FIGS. 3B and 3Cmay be adapted to implement the respective systems 180, 193.Applications and containers may be used to coordinate the cloud- andedge-based functionality, under the control of real-time orchestration.Other aspects of functionality or hardware configuration discussed forthe ECN nodes 150 may also be applicable to the edge computing node 193.The edge computing node 193 may implement control functions to control afield device.

Systems and techniques described herein may integrate “Mobile-edgeComputing” or “Multi-Access Edge Computing” (MEC) concepts, whichaccesses one or multiple types of Radio Access Networks (RANs) to allowincreases in speed for content, services, and applications. MEC allowsbase stations to act as intelligent service hubs, capable of deliveringhighly personalized services in edge networks. MEC provides proximity,speed, and flexible solutions to a variety of mobile devices, includingdevices used in next-generation SDIS operational environments. As anexample, a MEC approach is described in “Mobile-Edge Computing, A keytechnology towards 5G,” a paper published by the EuropeanTelecommunications Standards Institute (ETSI) as ETSI White Paper No.11, by Yun Chao Hu, et al., ISBN No. 979-10-92620-08-5, available athttp://www.etsi.org/news-events/news/1009-2015-09-news-new-white-paper-etsi-s-mobile-edge-computing-initiative-explained,which is incorporated herein in its entirety. It will be understood thatother aspects of 5G/next generation wireless networks, software-definednetworks, and network function virtualization, may be used with thepresent SIDS operational architecture.

FIG. 4 illustrates an example configuration 400 of a real-time servicebus (e.g., a configuration of the control messages bus 112) used withinan SDIS operational architecture. This configuration allows support forvarious processing control nodes, as discussed herein. For instance, thecontrol messages bus 112 may be used to connect respective controlprocessing nodes 410 (including various hardware and softwareimplementations on nodes 410A, 410B, 410C) and cloud-based services orcontrol server(s) 130A with various edge devices 420 (e.g., I/Ocontrollers 150, 160, 165, or edge computing nodes 191, 193, 195).

In an example, the control messages bus 112 may be implemented tosupport packet level, deterministic, control networks, with ratemonotonic control requirements. These features have conventionally beenprovided by proprietary Distributed Control System (DCS), SupervisoryControl And Data Acquisition (SCADA) or Programmable Logic Controller(PLC) components. Most of these systems were engineered to designparameters that limited the number of nodes and data elements withlittle ability to dynamically manage the quantity and quality of thedata for what is commonly a closed and isolated network within thefacility. Over the lifecycle of these systems, the desire to implementemerging new use cases has been severely limited by the underlyinginflexibility and limited scalability of expensive control systeminfrastructure.

With prior approaches, both open source and open standards-based servicebus middleware options have matured to the point that the criticalmission ecosystem of solution providers have embraced these technologiesas “best-in-breed” capabilities to build scalable, highly redundant,fault tolerant, real-time systems at a fraction of the historical cost.This has sparked a realization of new use cases that may be achieved forboth discrete and continuous processing where commodity level hardwareand open source, standards based software have converged to enablereal-time compute methods, while maintaining service orientedarchitecture based design principles.

In an example, control messages bus technologies may be extended furtherby enabling real-time compute at the hardware level by enabling TimeSensitive Networking (TSN) and Time Coordinated Compute (TCC) bothbetween and within platform nodes of a network. Both proprietary andopen standard-based solutions may be integrated with commodity hardwareenabled enhancements, including utilizing industry standards offered bythe OPC-UA (OPC Unified Architecture) and DDS (Data DistributionService) groups, and proprietary implementations like the SERCOSstandards where hard real-time requirements for discrete motion controlare mandatory in robotic and machine control applications.

In an example, the control messages bus and the overall SDISarchitecture may also be integrated with the Industrial InternetConsortium (IIC) features. These may include various formulating andtesting standards for the industrial use of TSN, which may enhance theperformance and QoS of both DDS and OPC-UA based solutions bydramatically reducing both packet level latency and jitter. Further,aspects of Object Management Group (OMG) and the OPC Foundationstandards may be positioned to support increased integration of OPC-UAand DDS implementation models that leverage the information modeling ofOPC-UA, and the QoS and performance capabilities of DDS in architecturaldesign. New use cases may include analytics and autonomous capabilities.

In an example, the SDIS architecture may be integrated with the use ofSoftware Defined Networking (SDN) features. SDN is a movement towards asoftware programmable network that separates the control plane from thedata plane to make the network and network functions more flexible,agile, scalable, and less dependent on networking equipment, vendors,and service providers. Two key use cases of SDN relevant to SDISinclude: service function chaining, which allows dynamic insertion ofintrusion detection/prevention functions, and dynamic reconfiguration torespond to events such as larger scale outages such as zone maintenance,natural disasters, etc. Further, the SDIS architecture may be integratedwith an SDN controller to control virtual switches using networkingprotocols such as Open vSwitch Database Management Protocol (OVSDB).Other use cases of SDN features may involve dynamic networkconfigurations, monitoring, and the abstraction of network functions invirtualized and dynamic systems.

FIG. 5A illustrates a first network configuration 500 for an exampledeployment of SDIS subsystems. The first network configuration 500illustrates a scaled-down, small-footprint deployment option thatcombines controller, storage, and compute functionality on a redundantpair of hosts (nodes 510A, 510B). In this configuration, the controllerfunctionality (for control applications or implementations) isactive/standby across the nodes 510A, 510B while the computefunctionality (for all remaining processes) is active/active, meaningthat VMs may be deployed to perform compute functionality on eitherhost.

For example, LVM/iSCSI may be used as the volume backend that isreplicated across the compute nodes, while each node also has a localdisk for ephemeral storage. Processor bandwidth and memory may be alsoreserved for the controller function. This two-node solution may providea lower cost and lower footprint solution when less processing andredundancy is needed.

FIG. 5B illustrates a second network configuration for deployment ofSDIS subsystems. The second network configuration 550 may providededicated storage nodes with high capacity, scalability, andperformance. As compared with the first network configuration 500, thesecond network configuration 550 allows controller, storage, and computefunctionalities to be deployed on separate physical hosts, allowingstorage and compute capacity to scale independently from each other.

In an example, the second network configuration may be provided from aconfiguration of up to eight storage nodes (nodes 530A-530N) and eightdisks per storage node in a high availability (e.g., Ceph) cluster(e.g., coordinated by controller nodes 520A, 520B), with the highavailability cluster providing image, volume, and objects storage forthe compute nodes. For instance, up to 100 compute nodes (e.g., node540) may be supported, each with its own local ephemeral storage for useby VMs. As will be understood, a variety of other network configurationsmay be implemented with use of the present SDIS architecture.

The SDIS architecture and accompanying data flows, orchestrations, andother features extended below, may also utilize aspects of MachineLearning, Cognitive Computing and Artificial Intelligence. For instance,The SDIS architecture may be integrated with a reference platform withfoundations in hardware-based security, interoperable services, andopen-source projects, including the use of big data analytics andmachine learning for cybersecurity. The SDIS architecture may utilizeimmutable hardware elements to prove device trust, and characterizenetwork traffic behavior based on filters augmented with machinelearning to separate bad traffic from benign.

The various components of the SDIS architecture may be integrated with arich set of security capabilities to enable an interoperable and secureindustrial system within real-world industrial settings. For example,such security capabilities may include hardware-based roots of trust,trusted execution environments, protected device identity,virtualization capabilities, and cryptographic services upon which arobust, real-time security architecture may be founded. Theconfiguration and functionality of such components within a functionalSDIS architecture deployment is further discussed in the followingsections.

Overview of Functional Orchestration

FIG. 6 illustrates an example of dynamically established set oforchestration operations 600 with use of a Composable Application SystemLayer (CSL) in a SDIS operational architecture. The CSL may be utilizedto enable a secure design and orchestration of control functions andapplications to support industrial operations.

In an example, the CSL maintains a library 680 of functional blocks 690,each representing control-loop logic and application components. Eachfunctional block may be interoperable with other functional blocks. Afunctional block may have multiple implementations, making it portable,such that it may operate on various platform architectures and leveragespecial features if available (e.g. hardware accelerators). In anexample, the CSL provides a control function for a cluster of edge nodes(e.g., ECNs); in further examples, the CSL provides control for VMs inthe control server or other computation points in the SDIS operationalarchitecture.

In an example, a process engineer (or other operator) defines controlflows and applications by combining and configuring existing functionalblocks 690 from the library 680. These functional blocks 690 mayrepresent application logic or control loops (e.g., control loops 670,data storage, analytics modules, data acquisition or actuation modules,or the like), control modules, or any other computation elements.Because these functional blocks 690 are reusable and interoperable, newcode needs to be written only when new functional blocks are required.In further examples, such functional blocks may be utilized to implementend-to-end logic, including control flows or end-to-end applicationsusing a graphical, drag-and-drop environment.

Starting from this application design, the CSL generates anorchestration plan 640 that specifies the required functional blocks andthe requirements for points of computation to execute those functionalblocks. As discussed in the following sections, orchestration 620 mayencompass the process of mapping the orchestration plan 640 to availablecompute and communication resources. The orchestration 620 may befurther adapted based on control standards design 610 (e.g., to conformthe resulting orchestration to various control laws, standards, orrequirements).

In an example, the CSL maintains a map 630 of computing and controlresources across the SDIS network. The map 630 comprehends the topologyof various compute points, from virtual machines in a data center tocontrol points and the attached sensors and actuators. The map 630 alsoincludes the hardware capabilities and dynamic characteristics of thecontrol points. The map is updated regularly, allowing the system toconstantly adapt to component failures. The orchestration 620 and thecontrol loop 670 communicate using monitoring logic 650 and functiondeployments 660. The monitoring logic 650 outputs information from afield device or the control loop 670, which is used as an input to themap 630. The function deployment 660 is used as an input or statesetting for the control loop 670.

When an operator deploys a new application definition (e.g., theorchestration 620 receives an output from the control standards design610), the orchestration 620 determines how to best fit the functionalblocks 690 to the set of available resources in map 630, and deploys theunderlying software components that implement the functional blocks 690.Deployment of an end-to-end application may include, for example,creating virtual machines within a server, injecting code into controlloops (e.g., control loops 670), and creating communication pathsbetween components, as needed. Orchestration 620 also may be dynamic toallow functional blocks to be migrated upon failure of a computationalresource, without requiring a system-wide restart. In addition, updatesto the implementation of a component may be pushed, causing code to beupdated as needed.

The CSL may also incorporate security and privacy features, such as toestablish trust with participating devices (including edge nodes or acontrol server). In further examples, the CSL may be integrated withkey-management used for onboarding new devices and revoking obsoletedevices. The CSL may deliver keys to function blocks 660 to enablesecure communication with other function blocks 660. The CSL may alsodeliver secured telemetry and control, integrity and isolated executionof deployed code, and integrity of communication among functional blocks690.

Orchestration technologies today predominantly execute by function,application, virtual machine, or container technology. However, inherentdependencies between distributed applications are not generally managedin low-latency, high frequency mission-critical timeframes for controlstrategy implementations today. For embedded systems in general, dynamicorchestration historically has not been applied due to the technicallimitations of managing application dependencies at runtime.

In an example, features of an SDIS architecture may be adapted tosupport the holistic orchestration and management of multiple dependentapplications (function blocks) that execute across a distributedresource pool, to enable orchestration at an embedded control strategylevel in a distributed system configuration. This provides a controlstrategy orchestration capability to operational technology environmentswhile elevating overall system performance at an expected reduced totalcost. For instance, an example orchestration method may incorporatedynamic network discovery, resource simulation in advance of anyorchestration action, and simulation coupled with global resourceoptimization and prediction utilized as part of an orchestrator rule setdecision tree.

The distributed resource pool may encompass applications that span: (a)a single application running in a single native device, where a secondredundant application is available on an additional native device; (b)multiple coordinated applications running in multiple native devices;(c) multiple coordinated applications running in a single virtualmachine, where the virtual machine is running on a single embeddeddevice or server; (d) multiple coordinated applications running acrossmultiple virtual machines, where each virtual machine runs in adedicated embedded device or server; (e) multiple coordinatedapplications that span multiple containers contained in one virtualmachine, where the virtual machine runs in a dedicated embedded deviceor server; or (f) multiple coordinated applications spanning multiplecontainers, where the containers are running on multiple embeddeddevices or servers. Any mixture of these application scenarios may alsoapply.

In an example, orchestration may include measurement of resources orreservation of resources, such as compute resources on a node (e.g., onthe CPU or special purpose compute blocks like an FPGA or GPU),particular device capabilities (access to a sensor/actuator, securitydevice (e.g., TPM), pre-installed software), storage resources on a node(memory or disk), network resources (latency or bandwidth, perhapsguaranteed via TSN), or the like.

An extended orchestrator rule set may be defined to include criteriabeyond standard compute, storage, and memory metrics, such as to specifyapplication cycle time, application runtime, application input/outputsignal dependency, or application process sequencing (e.g. a mandatorysequence that specifies which application(s) runs before or after otherapplication blocks). This orchestration technique may provide theability, at a distributed application control strategy level, toleverage lower cost commodity hardware and software to achieve bettersystem performance at a control strategy level, while enabling newlevels of system redundancy and failover at a lower cost across multipleapplications running in ISA levels L1-L3. Further, orchestrationsensitivity at the broader control strategy level may enable new levelsof high availability for embedded systems at a lower cost. This mayresult in an increase of general system and application uptime fororchestrated and coordinated control applications, while reducingunplanned downtime for production operations at a higher ISA level thanavailable with conventional approaches.

The following orchestration techniques may also enable additionalmaintenance tasks to occur (without production downtime) for systemswhere system redundancy is designed into the automation configuration.These techniques enable increased interoperability for where controlstrategies execute among vendor hardware where platform agnosticvirtualization and containerization is leveraged. These techniques alsoleverage current, historical and simulation results to optimize workloadplacement for operational technology environments for real-timeoperations. Further, these techniques may leverage predictions of futureorchestration events to pre-plan workload placement.

In an example, a distributed resource pool is defined as a combinationof compute, storage, memory across networked computing assets with theaddition of function block scheduling frequency, before and afterprocessing assignments, latency tolerance for the purpose of executingapplication control strategies. For instance, a control strategy (orapplication), may be defined by a physically distributed, coordinatedset of building blocks with very strict time, block-to-block scheduling,and run-time requirements for execution. The orchestration of thesebuilding blocks in time is coordinated with respect to the order ofexecution, processing latency and full execution cycle of all buildingblocks that make up the overall application control strategy.

Scalable Edge Compute in a Distributed Control Environment

Current solutions require the end user to estimate the amount of computerequired, and add additional compute capability to future proof thedeployment. These approaches waste money, electrical, and thermalenergy. This also risks the over provisioned compute becoming oldtechnology before the compute is actually needed.

The techniques discussed herein allow a high performance CPU to beactivated, from an initial dormant or inactive state, in an edge controlnode of an industrial control system by a centralized orchestrationsystem that understands the CPU performance needs of the controlstrategy of the industrial system. Initial customer investment is low,as each edge control node is initially sold as a low cost, lowperformance device. Only the required compute (right sized compute) ispurchased and provisioned, which optimizes the monetary investment,thermal footprint and electrical energy consumption. This solutionprovides an expandable compute footprint in the control system.

FIG. 7 illustrates an industrial control system (ICS) Ring Topologynetwork 702.

An industrial control system is generally made up of Programmable LogicController 704, Remote 10 (RIO) (e.g., 706) and Field Devices (e.g.,708). A typical deployment may consist of rings of Remote IO unitscontrolled by a PLC 704. IO and field compute are typically locked inPLC 704 (e.g., at FIG. 7).

FIG. 8 illustrates an edge control topology network. The edge controltopology network includes an orchestration server 802 (e.g., asdescribed above for orchestration 620), a bridge 804, a plurality ofedge control nodes (e.g., ECN 806), and one or more field devices (e.g.,808). The orchestration server 802 is used to provision, control, ororchestrate actions at the ECN devices (e.g., 806), which are connectedfor example in a ring network to each other, and to the orchestrationserver 802 via the bridge 804).

One way that SDIS improves the functioning of a system is thedistribution of control functionality across an ICS. The orchestrationserver 802 may be used to control the edge control node 806, whichincludes the option of performing both IO and Compute on a single deviceand uses Orchestration services to distribute workloads to the bestavailable resource.

Typically the ring of edge control nodes (ECNs) may be deployed inthermally constrained environments, for example, cabinets with zeroairflow or unregulated temperatures. In an example, there may be up to96 IO in a single cabinet, which means up to 96 ECNs. This may prohibiteach ECN from including both IO and High Performance compute, as thehigh performance compute device will generate excessive heat and raisethe ambient temperature above the safe operating level of the ECNs.Additionally, a high performance processor may not be needed at everyECN when there is not a high compute demand of the control system.Therefore, the systems and techniques described herein provide acapability to install just the compute resources that are needed toexecute the control strategy, and to not exceed cost and power targets,while still allowing for changes in each ECN. Thus, in an example, notevery ECN has a high performance processor or high control capabilities.

FIG. 9 illustrates an edge control node (ECN) block diagram 902. In anexample, the following techniques provide a “right size” provisioning ofa compute problem with the introduction of a compute scalable ECN asshown in FIG. 9.

The primary ingredient of the ECN 902 may be a system on chip 904, whichhas both higher performance compute (e.g., CPU) 906 and a microprocessor(MCU) 908 for low performance compute. The MCU 908 may be used toconvert IO data coming from the IO Subsystem 912 to a network component910, such as an Ethernet TSN based middleware such as OPCUA Pub/Sub orDDS. The ECN 902 may be delivered to customers with the High PerformanceCPU 906 in an inactive state. For example, the High Performance CPU 906may not be accessible for use in the inactive state, such as until aspecial “activation signal” is sent to the High Performance CPU 906, forexample from an orchestrator (e.g., the orchestrator may send a signalsent to the MCU 908 to activate the CPU 906).

The ECN 902 may be initially installed as a low cost, low power devicefor IO conversion using the MCU 908. For example, the High PerformanceCPU 906 is initially disabled, and initially the ECN 902 includes theSoC 904 and IO Subsystem 912 activated, without high controlcapabilities. The high performance processor 906 may be inactive, withthe ECN 902 only allowing IO Conversion initially, in an example.

FIG. 10 illustrates an ECN-based ring topology diagram. FIG. 10 showshow a scalable compute ECN may fit into the classic ring topology. FIG.10 further shows an initial state of deployment, where all highperformance CPUs are disabled. As shown in FIG. 10 each ECN has theability to convert IO to a data bus standard, but no real capability toexecute control functions.

In an example, after deployment, the orchestration server 802 maydetermine how many high performance CPUs are needed, and then send acode to activate one or more CPUs using respective MCUs at particularECNs. The orchestration server 802 may provide a cost/benefit analysisas part of the scheduling function performed by the orchestration server802. In an example, a fee may be charged to activate the CPU 906, suchas according to a schedule, such as monthly, yearly licenses, or thelike. The CPU 906 may be activated or deactivated according to need(e.g., as determined by the orchestrator or the user). The limitedlicense may be cheaper than full deployment. In another example, onceactivated, the CPU 906 may remain activated indefinitely (e.g.,activated permanently for a one-time fee).

In an example, not activating the CPU 906 may reduce thermal output.This may be controlled separately from any fee schedules. For example,once activated, the CPU 906 may be deactivated or moved to a low powerstate to save on thermal output (even in an example where the CPU 906was permanently activated). The CPU 906 may execute control instructionsin a high power state and move to a low power state when execution iscompleted.

In an example, an activation code may be a special packet, sent to theMCU 908. The activation code may be evaluated for validity by the MCU908 including determining how long the code is good for, etc. The MCU908 may send an activation signal directly to the CPU (e.g., afterreceiving a signal from an orchestrator).

The MCU 908 may turn on power rails, boot the CPU 906, download latestfirmware, etc., when activating the CPU 906 from the inactive state. Inan example, the CPU 906 may have a low or high power mode, which may beactivated or deactivated instead of turning the CPU 906 off or on. Thisexample may be useful in cases where the CPU 906 is put in a low powerstate instead of being powered off to reduce thermal output, such aswhen the CPU 906 may be needed to be activated quickly.

In an example, the low power state may be implemented by providingcryptographic tokens that the orchestrator 802 obtains from the CPUmanufacturer. These tokens may be sent to the CPU 906 via the MCU 908.The tokens may, for example, be signed using a key that only the CPUmanufacturer and the CPU 906 know (e.g., burned into CPU 906 atmanufacture), allowing each token to be validated. Each token may beunique, allowing the CPU 906 to run for some amount of time.

In another example, the tokens are authenticated by the MCU 908 using asecret known to the manufacturer and the MCU 908. For example, as longas the MCU 908 and the CPU 906 are manufactured together in a singlepackage of an SoC. This example may prevent a denial of service attackcreated by having the CPU 906 woken up to validate the token.

FIG. 11 illustrates data flow through an ECN-based ring topology. In anexample, the orchestration system 802 analyzes the control strategy tounderstand how much compute is required to satisfy the compute needs ofthe control strategy. Once the orchestration system has generated thecompute requirements, the end user may purchase the required amount ofHigh Performance CPU activation codes from the ECN vendor. Theorchestration system 802 will send the authenticated activation codes tospecified ECNs in the array of ECNs, which enables the computeresources. This flow is shown in FIG. 11.

The process of enabling compute need not be a one-time event. As thecomplexity of the control strategy increases and compute demandsincrease, the end user may continue to purchase and activate morecompute resources (or deactivate CPU resources when not needed). Forexample, the orchestrator may send a deactivation signal to an ECN todeactivate a CPU at that ECN. The ECN vendor may implement a temporalservice model, where the end user buys activation licenses on a monthlyor yearly basis. This models also allows the end users to let theactivation codes expire, allowing some of the compute resources to goback into low power dormant state saving the recurring fees.

FIG. 12 illustrates a flowchart 1200 of a method for activating a CPU(e.g., of an ECN) according to an example. Flowchart 1200 includes anoperation 1210 to determine, at an orchestration server, computationalrequirements of edge control nodes in an industrial control system(e.g., a ring deployment). Flowchart 1200 includes an operation 1220 toreceive an indication to activate CPUs of one or more edge control nodesor determine that one or more CPUs need to be activated. Flowchart 1200includes an operation 1230 to send authenticated activation codes to theedge control nodes with CPUs to be activated. In an example, operations1210-1230 (above) may be performed by the orchestration server, andoperations 1240-1270 (below) may be performed by an ECN. A method usingthe flowchart 1200 may include performing operations 1210-1230 or1240-1270 or both. Flowchart 1200 includes an operation 1240 to receivean authenticated activation code at an edge control node. Flowchart 1200includes an operation 1250 to authenticate the code at the edge controlnode (e.g., at the CPU). Flowchart 1200 includes an operation 1260 toactivate a CPU of the edge control node using a MCU (low performanceprocessor). Flowchart 1200 includes an optional operation 1270 toreceive an update at the edge control node from the orchestration serverto deactivate the CPU or place the CPU in a low power state. In anexample, the ECN may be part of a ring network of an industrial controlsystem.

FIG. 13 illustrates a flowchart 1300 of a method for activating a CPUaccording to an example. The operations of flowchart 1300 may beperformed by an orchestration server. The orchestration server may becommunicatively coupled to a ring network of edge control nodes, such asvia a bridge device.

The flowchart 1300 includes an optional operation 1310 to determinecomputational requirements of edge control nodes in an industrialcontrol system. In an example, the edge control nodes may be nodes in aring topology network with a bridge device connecting the network to theorchestration server.

The flowchart 1300 includes an operation 1320 to receive IO data via abridge connecting an orchestration server to an edge control node. TheIO data may be converted at a microcontroller (MCU) of the edge controlnode from data generated at an IO subsystem. The conversion may be to apacket sent by an Ethernet switch of a system on a chip of the edgecontrol node (which may include the MCU as well). In another example,the data converted by the MCU may be data generated by the MCU itself,such as a power state of the field device or the edge control node.

The flowchart 1300 includes an operation 1330 to send an authenticatedactivation code to the edge control node to activate a CPU of the edgecontrol node, with this CPU initially in an inactivated state. In anexample, the authenticated activation code is authenticated by the MCUbefore the CPU is activated.

The flowchart 1300 includes an operation 1340 to send processinginstructions to the CPU for execution.

The flowchart 1300 includes an optional operation 1350 to send adeactivation code to the edge control node to deactivate the CPU of theedge control node.

The method may include an operation to determine computationalrequirements of edge control nodes in an industrial control systemincluding the edge control node. In an example, the CPU is activatedbased on a determination by the orchestration server that the CPU is tobe activated to satisfy a control strategy for the industrial controlsystem. In another example, the orchestration server may receive anindication to activate the CPU of the edge control node of the edgecontrol nodes.

IoT Devices and Networks

The techniques described above may be implemented in connection with avariety of device deployments, including in those of any number of IoTnetworks and topologies. Accordingly, it will be understood that variousembodiments of the present techniques may involve the coordination ofedge devices, the fog and intermediary devices, and cloud entities amongheterogeneous and homogeneous networks. Some of the example topologiesand arrangements of such networks are provided in the followingparagraphs.

FIG. 14 illustrates an example domain topology for respectiveinternet-of-things (IoT) networks coupled through links to respectivegateways. The internet of things (IoT) is a concept in which a largenumber of computing devices are interconnected to each other and to theInternet to provide functionality and data acquisition at very lowlevels. Thus, as used herein, an IoT device may include a semiautonomousdevice performing a function, such as sensing or control, among others,in communication with other IoT devices and a wider network, such as theInternet.

IoT devices are physical objects that may communicate on a network, andmay include sensors, actuators, and other input/output components, suchas to collect data or perform actions from a real world environment. Forexample, IoT devices may include low-powered devices that are embeddedor attached to everyday things, such as buildings, vehicles, packages,etc., to provide an additional level of artificial sensory perception ofthose things. Recently, IoT devices have become more popular and thusapplications using these devices have proliferated.

Often, IoT devices are limited in memory, size, or functionality,allowing larger numbers to be deployed for a similar cost to smallernumbers of larger devices. However, an IoT device may be a smart phone,laptop, tablet, or PC, or other larger device. Further, an IoT devicemay be a virtual device, such as an application on a smart phone orother computing device. IoT devices may include IoT gateways, used tocouple IoT devices to other IoT devices and to cloud applications, fordata storage, process control, and the like.

Networks of IoT devices may include commercial and home automationdevices, such as water distribution systems, electric power distributionsystems, pipeline control systems, plant control systems, lightswitches, thermostats, locks, cameras, alarms, motion sensors, and thelike. The IoT devices may be accessible through remote computers,servers, and other systems, for example, to control systems or accessdata.

The future growth of the Internet and like networks may involve verylarge numbers of IoT devices. Accordingly, in the context of thetechniques discussed herein, a number of innovations for such futurenetworking will address the need for all these layers to growunhindered, to discover and make accessible connected resources, and tosupport the ability to hide and compartmentalize connected resources.Any number of network protocols and communications standards may beused, wherein each protocol and standard is designed to address specificobjectives. Further, the protocols are part of the fabric supportinghuman accessible services that operate regardless of location, time orspace. The innovations include service delivery and associatedinfrastructure, such as hardware and software; security enhancements;and the provision of services based on Quality of Service (QoS) termsspecified in service level and service delivery agreements. As will beunderstood, the use of IoT devices and networks, such as thoseintroduced in the system examples discussed above, present a number ofnew challenges in a heterogeneous network of connectivity comprising acombination of wired and wireless technologies.

FIG. 14 specifically provides a simplified drawing of a domain topologythat may be used for a number of internet-of-things (IoT) networkscomprising IoT devices 1404, with the IoT networks 1456, 1458, 1460,1462, coupled through backbone links 1402 to respective gateways 1454.For example, a number of IoT devices 1404 may communicate with a gateway1454, and with each other through the gateway 1454. To simplify thedrawing, not every IoT device 1404, or communications link (e.g., link1416, 1422, 1428, or 1432) is labeled. The backbone links 1402 mayinclude any number of wired or wireless technologies, including opticalnetworks, and may be part of a local area network (LAN), a wide areanetwork (WAN), or the Internet. Additionally, such communication linksfacilitate optical signal paths among both IoT devices 1404 and gateways1454, including the use of MUXing/deMUXing components that facilitateinterconnection of the various devices.

The network topology may include any number of types of IoT networks,such as a mesh network provided with the network 1456 using Bluetoothlow energy (BLE) links 1422. Other types of IoT networks that may bepresent include a wireless local area network (WLAN) network 1458 usedto communicate with IoT devices 1404 through IEEE 802.11 (Wi-Fi®) links1428, a cellular network 1460 used to communicate with IoT devices 1404through an LTE/LTE-A (4G) or 5G cellular network, and a low-power widearea (LPWA) network 1462, for example, a LPWA network compatible withthe LoRaWan specification promulgated by the LoRa alliance, or a IPv6over Low Power Wide-Area Networks (LPWAN) network compatible with aspecification promulgated by the Internet Engineering Task Force (IETF).Further, the respective IoT networks may communicate with an outsidenetwork provider (e.g., a tier 2 or tier 3 provider) using any number ofcommunications links, such as an LTE cellular link, an LPWA link, or alink based on the IEEE 802.15.4 standard, such as Zigbee®. Therespective IoT networks may also operate with use of a variety ofnetwork and internet application protocols such as ConstrainedApplication Protocol (CoAP). The respective IoT networks may also beintegrated with coordinator devices that provide a chain of links thatforms cluster tree of linked devices and networks.

Each of these IoT networks may provide opportunities for new technicalfeatures, such as those as described herein. The improved technologiesand networks may enable the exponential growth of devices and networks,including the use of IoT networks into as fog devices or systems. As theuse of such improved technologies grows, the IoT networks may bedeveloped for self-management, functional evolution, and collaboration,without needing direct human intervention. The improved technologies mayeven enable IoT networks to function without centralized controlledsystems. Accordingly, the improved technologies described herein may beused to automate and enhance network management and operation functionsfar beyond current implementations.

In an example, communications between IoT devices 1404, such as over thebackbone links 1402, may be protected by a decentralized system forauthentication, authorization, and accounting (AAA). In a decentralizedAAA system, distributed payment, credit, audit, authorization, andauthentication systems may be implemented across interconnectedheterogeneous network infrastructure. This allows systems and networksto move towards autonomous operations. In these types of autonomousoperations, machines may even contract for human resources and negotiatepartnerships with other machine networks. This may allow the achievementof mutual objectives and balanced service delivery against outlined,planned service level agreements as well as achieve solutions thatprovide metering, measurements, traceability and trackability. Thecreation of new supply chain structures and methods may enable amultitude of services to be created, mined for value, and collapsedwithout any human involvement.

Such IoT networks may be further enhanced by the integration of sensingtechnologies, such as sound, light, electronic traffic, facial andpattern recognition, smell, vibration, into the autonomous organizationsamong the IoT devices. The integration of sensory systems may allowsystematic and autonomous communication and coordination of servicedelivery against contractual service objectives, orchestration andquality of service (QoS) based swarming and fusion of resources. Some ofthe individual examples of network-based resource processing include thefollowing.

The mesh network 1456, for instance, may be enhanced by systems thatperform inline data-to-information transforms. For example, self-formingchains of processing resources comprising a multi-link network maydistribute the transformation of raw data to information in an efficientmanner, and the ability to differentiate between assets and resourcesand the associated management of each. Furthermore, the propercomponents of infrastructure and resource based trust and serviceindices may be inserted to improve the data integrity, quality,assurance and deliver a metric of data confidence.

The WLAN network 1458, for instance, may use systems that performstandards conversion to provide multi-standard connectivity, enablingIoT devices 1404 using different protocols to communicate. Furthersystems may provide seamless interconnectivity across a multi-standardinfrastructure comprising visible Internet resources and hidden Internetresources.

Communications in the cellular network 1460, for instance, may beenhanced by systems that offload data, extend communications to moreremote devices, or both. The LPWA network 1462 may include systems thatperform non-Internet protocol (IP) to IP interconnections, addressing,and routing. Further, each of the IoT devices 1404 may include theappropriate transceiver for wide area communications with that device.Further, each IoT device 1404 may include other transceivers forcommunications using additional protocols and frequencies. This isdiscussed further with respect to the communication environment andhardware of an IoT processing device depicted in FIGS. 16 and 17.

Finally, clusters of IoT devices may be equipped to communicate withother IoT devices as well as with a cloud network. This may allow theIoT devices to form an ad-hoc network between the devices, allowing themto function as a single device, which may be termed a fog device. Thisconfiguration is discussed further with respect to FIG. 15 below.

FIG. 15 illustrates a cloud computing network in communication with amesh network of IoT devices (devices 1502) operating as a fog device atthe edge of the cloud computing network. The mesh network of IoT devicesmay be termed a fog 1520, operating at the edge of the cloud 1500. Tosimplify the diagram, not every IoT device 1502 is labeled.

The fog 1520 may be considered to be a massively interconnected networkwherein a number of IoT devices 1502 are in communications with eachother, for example, by radio links 1522. As an example, thisinterconnected network may be facilitated using an interconnectspecification released by the Open Connectivity Foundation™ (OCF). Thisstandard allows devices to discover each other and establishcommunications for interconnects. Other interconnection protocols mayalso be used, including, for example, the optimized link state routing(OLSR) Protocol, the better approach to mobile ad-hoc networking(B.A.T.M.A.N.) routing protocol, or the OMA Lightweight M2M (LWM2M)protocol, among others.

Three types of IoT devices 1502 are shown in this example, gateways1504, data aggregators 1526, and sensors 1528, although any combinationsof IoT devices 1502 and functionality may be used. The gateways 1504 maybe edge devices that provide communications between the cloud 1500 andthe fog 1520, and may also provide the backend process function for dataobtained from sensors 1528, such as motion data, flow data, temperaturedata, and the like. The data aggregators 1526 may collect data from anynumber of the sensors 1528, and perform the processing function for theanalysis. The results, raw data, or both may be passed along to thecloud 1500 through the gateways 1504. The sensors 1528 may be full IoTdevices 1502, for example, capable of both collecting data andprocessing the data. In some cases, the sensors 1528 may be more limitedin functionality, for example, collecting the data and allowing the dataaggregators 1526 or gateways 1504 to process the data.

Communications from any IoT device 1502 may be passed along a convenientpath (e.g., a most convenient path) between any of the IoT devices 1502to reach the gateways 1504. In these networks, the number ofinterconnections provide substantial redundancy, allowing communicationsto be maintained, even with the loss of a number of IoT devices 1502.Further, the use of a mesh network may allow IoT devices 1502 that arevery low power or located at a distance from infrastructure to be used,as the range to connect to another IoT device 1502 may be much less thanthe range to connect to the gateways 1504.

The fog 1520 provided from these IoT devices 1502 may be presented todevices in the cloud 1500, such as a server 1506, as a single devicelocated at the edge of the cloud 1500, e.g., a fog device. In thisexample, the alerts coming from the fog device may be sent without beingidentified as coming from a specific IoT device 1502 within the fog1520. In this fashion, the fog 1520 may be considered a distributedplatform that provides computing and storage resources to performprocessing or data-intensive tasks such as data analytics, dataaggregation, and machine-learning, among others.

In some examples, the IoT devices 1502 may be configured using animperative programming style, e.g., with each IoT device 1502 having aspecific function and communication partners. However, the IoT devices1502 forming the fog device may be configured in a declarativeprogramming style, allowing the IoT devices 1502 to reconfigure theiroperations and communications, such as to determine needed resources inresponse to conditions, queries, and device failures. As an example, aquery from a user located at a server 1506 about the operations of asubset of equipment monitored by the IoT devices 1502 may result in thefog 1520 device selecting the IoT devices 1502, such as particularsensors 1528, needed to answer the query. The data from these sensors1528 may then be aggregated and analyzed by any combination of thesensors 1528, data aggregators 1526, or gateways 1504, before being senton by the fog 1520 device to the server 1506 to answer the query. Inthis example, IoT devices 1502 in the fog 1520 may select the sensors1528 used based on the query, such as adding data from flow sensors ortemperature sensors. Further, if some of the IoT devices 1502 are notoperational, other IoT devices 1502 in the fog 1520 device may provideanalogous data, if available.

In an example, the various aspects of workload orchestration andoperations may be adapted to the various network topologies andapproaches depicted in FIG. 15. For example, a system may establish avariety of workloads executing in the cloud 1500 in coordination withthe IoT devices 1502. These workloads could be orchestrated in the cloud1500 or fog 1520 from the edge (e.g., from IoT devices 1502), or suchworkloads may be orchestrated on the edge by the cloud 1500 or the fog1520. Such concepts may also apply to gateways 1504 and data aggregators1526 and other devices and nodes within the network topology.

In other examples, the operations and functionality described above withreference to the systems described above may be embodied by an IoTdevice machine in the example form of an electronic processing system,within which a set or sequence of instructions may be executed to causethe electronic processing system to perform any one of the methodologiesdiscussed herein, according to an example. The machine may be an IoTdevice or an IoT gateway, including a machine embodied by aspects of apersonal computer (PC), a tablet PC, a personal digital assistant (PDA),a mobile telephone or smartphone, or any machine capable of executinginstructions (sequential or otherwise) that specify actions to be takenby that machine. Further, while only a single machine may be depictedand referenced in the example above, such machine shall also be taken toinclude any collection of machines that individually or jointly executea set (or multiple sets) of instructions to perform any one or more ofthe methodologies discussed herein. Further, these and like examples toa processor-based system shall be taken to include any set of one ormore machines that are controlled by or operated by a processor (e.g., acomputer) to individually or jointly execute instructions to perform anyone or more of the methodologies discussed herein.

FIG. 16 illustrates a drawing of a cloud computing network, or cloud1600, in communication with a number of Internet of Things (IoT)devices. The cloud 1600 may represent the Internet, or may be a localarea network (LAN), or a wide area network (WAN), such as a proprietarynetwork for a company. The IoT devices may include any number ofdifferent types of devices, grouped in various combinations. Forexample, a traffic control group 1606 may include IoT devices alongstreets in a city. These IoT devices may include stoplights, trafficflow monitors, cameras, weather sensors, and the like. The trafficcontrol group 1606, or other subgroups, may be in communication with thecloud 1600 through wired or wireless links 1608, such as LPWA links,optical links, and the like. Further, a wired or wireless sub-network1612 may allow the IoT devices to communicate with each other, such asthrough a local area network, a wireless local area network, and thelike. The IoT devices may use another device, such as a gateway 1610 or1628 to communicate with remote locations such as the cloud 1600; theIoT devices may also use one or more servers 1630 to facilitatecommunication with the cloud 1600 or with the gateway 1610. For example,the one or more servers 1630 may operate as an intermediate network nodeto support a local edge cloud or fog implementation among a local areanetwork. Further, the gateway 1628 that is depicted may operate in acloud-to-gateway-to-many edge devices configuration, such as with thevarious IoT devices 1614, 1620, 1624 being constrained or dynamic to anassignment and use of resources in the cloud 1600.

Other example groups of IoT devices may include remote weather stations1614, local information terminals 1616, alarm systems 1618, automatedteller machines 1620, alarm panels 1622, or moving vehicles, such asemergency vehicles 1624 or other vehicles 1626, among many others. Eachof these IoT devices may be in communication with other IoT devices,with servers 1604, with another IoT fog device or system (not shown, butdepicted in FIG. 15), or a combination therein. The groups of IoTdevices may be deployed in various residential, commercial, andindustrial settings (including in both private or public environments).

As may be seen from FIG. 16, a large number of IoT devices may becommunicating through the cloud 1600. This may allow different IoTdevices to request or provide information to other devices autonomously.For example, a group of IoT devices (e.g., the traffic control group1606) may request a current weather forecast from a group of remoteweather stations 1614, which may provide the forecast without humanintervention. Further, an emergency vehicle 1624 may be alerted by anautomated teller machine 1620 that a burglary is in progress. As theemergency vehicle 1624 proceeds towards the automated teller machine1620, it may access the traffic control group 1606 to request clearanceto the location, for example, by lights turning red to block crosstraffic at an intersection in sufficient time for the emergency vehicle1624 to have unimpeded access to the intersection.

Clusters of IoT devices, such as the remote weather stations 1614 or thetraffic control group 1606, may be equipped to communicate with otherIoT devices as well as with the cloud 1600. This may allow the IoTdevices to form an ad-hoc network between the devices, allowing them tofunction as a single device, which may be termed a fog device or system(e.g., as described above with reference to FIG. 15).

FIG. 17 is a block diagram of an example of components that may bepresent in an IoT device 1750 for implementing the techniques describedherein. The IoT device 1750 may include any combinations of thecomponents shown in the example or referenced in the disclosure above.The components may be implemented as ICs, portions thereof, discreteelectronic devices, or other modules, logic, hardware, software,firmware, or a combination thereof adapted in the IoT device 1750, or ascomponents otherwise incorporated within a chassis of a larger system.Additionally, the block diagram of FIG. 17 is intended to depict ahigh-level view of components of the IoT device 1750. However, some ofthe components shown may be omitted, additional components may bepresent, and different arrangement of the components shown may occur inother implementations.

The IoT device 1750 may include a processor 1752, which may be amicroprocessor, a multi-core processor, a multithreaded processor, anultra-low voltage processor, an embedded processor, or other knownprocessing element. The processor 1752 may be a part of a system on achip (SoC) in which the processor 1752 and other components are formedinto a single integrated circuit, or a single package, such as theEdison™ or Galileo™ SoC boards from Intel. As an example, the processor1752 may include an Intel® Architecture Core™ based processor, such as aQuark™, an Atom™, an i3, an i5, an i7, or an MCU-class processor, oranother such processor available from Intel® Corporation, Santa Clara,Calif. However, any number other processors may be used, such asavailable from Advanced Micro Devices, Inc. (AMD) of Sunnyvale, Calif.,a MIPS-based design from MIPS Technologies, Inc. of Sunnyvale, Calif.,an ARM-based design licensed from ARM Holdings, Ltd. or customerthereof, or their licensees or adopters. The processors may includeunits such as an A5-A10 processor from Apple® Inc., a Snapdragon™processor from Qualcomm® Technologies, Inc., or an OMAP™ processor fromTexas Instruments, Inc.

The processor 1752 may communicate with a system memory 1754 over aninterconnect 1756 (e.g., a bus). Any number of memory devices may beused to provide for a given amount of system memory. As examples, thememory may be random access memory (RAM) in accordance with a JointElectron Devices Engineering Council (JEDEC) design such as the DDR ormobile DDR standards (e.g., LPDDR, LPDDR2, LPDDR3, or LPDDR4). Invarious implementations the individual memory devices may be of anynumber of different package types such as single die package (SDP), dualdie package (DDP) or quad die package (Q17P). These devices, in someexamples, may be directly soldered onto a motherboard to provide a lowerprofile solution, while in other examples the devices are configured asone or more memory modules that in turn couple to the motherboard by agiven connector. Any number of other memory implementations may be used,such as other types of memory modules, e.g., dual inline memory modules(DIMMs) of different varieties including but not limited to microDIMMsor MiniDIMMs.

To provide for persistent storage of information such as data,applications, operating systems and so forth, a storage 1758 may alsocouple to the processor 1752 via the interconnect 1756. In an examplethe storage 1758 may be implemented via a solid state disk drive (SSDD).Other devices that may be used for the storage 1758 include flash memorycards, such as SD cards, microSD cards, xD picture cards, and the like,and USB flash drives. In low power implementations, the storage 1758 maybe on-die memory or registers associated with the processor 1752.However, in some examples, the storage 1758 may be implemented using amicro hard disk drive (HDD). Further, any number of new technologies maybe used for the storage 1758 in addition to, or instead of, thetechnologies described, such resistance change memories, phase changememories, holographic memories, or chemical memories, among others.

The components may communicate over the interconnect 1756. Theinterconnect 1756 may include any number of technologies, includingindustry standard architecture (ISA), extended ISA (EISA), peripheralcomponent interconnect (PCI), peripheral component interconnect extended(PCIx), PCI express (PCIe), or any number of other technologies. Theinterconnect 1756 may be a proprietary bus, for example, used in a SoCbased system. Other bus systems may be included, such as an I2Cinterface, an SPI interface, point to point interfaces, and a power bus,among others.

The interconnect 1756 may couple the processor 1752 to a meshtransceiver 1762, for communications with other mesh devices 1764. Themesh transceiver 1762 may use any number of frequencies and protocols,such as 2.4 Gigahertz (GHz) transmissions under the IEEE 802.15.4standard, using the Bluetooth® low energy (BLE) standard, as defined bythe Bluetooth® Special Interest Group, or the ZigBee® standard, amongothers. Any number of radios, configured for a particular wirelesscommunication protocol, may be used for the connections to the meshdevices 1764. For example, a WLAN unit may be used to implement Wi-Fi™communications in accordance with the Institute of Electrical andElectronics Engineers (IEEE) 802.11 standard. In addition, wireless widearea communications, e.g., according to a cellular or other wirelesswide area protocol, may occur via a WWAN unit.

The mesh transceiver 1762 may communicate using multiple standards orradios for communications at different range. For example, the IoTdevice 1750 may communicate with close devices, e.g., within about 10meters, using a local transceiver based on BLE, or another low powerradio, to save power. More distant mesh devices 1764, e.g., within about50 meters, may be reached over ZigBee or other intermediate powerradios. Both communications techniques may take place over a singleradio at different power levels, or may take place over separatetransceivers, for example, a local transceiver using BLE and a separatemesh transceiver using ZigBee.

A wireless network transceiver 1766 may be included to communicate withdevices or services in the cloud 1700 via local or wide area networkprotocols. The wireless network transceiver 1766 may be a LPWAtransceiver that follows the IEEE 802.15.4, or IEEE 802.15.4g standards,among others. The IoT device 1750 may communicate over a wide area usingLoRaWAN™ (Long Range Wide Area Network) developed by Semtech and theLoRa Alliance. The techniques described herein are not limited to thesetechnologies, but may be used with any number of other cloudtransceivers that implement long range, low bandwidth communications,such as Sigfox, and other technologies. Further, other communicationstechniques, such as time-slotted channel hopping, described in the IEEE802.15.4e specification may be used.

Any number of other radio communications and protocols may be used inaddition to the systems mentioned for the mesh transceiver 1762 andwireless network transceiver 1766, as described herein. For example, theradio transceivers 1762 and 1766 may include an LTE or other cellulartransceiver that uses spread spectrum (SPA/SAS) communications forimplementing high speed communications. Further, any number of otherprotocols may be used, such as Wi-Fi® networks for medium speedcommunications and provision of network communications.

The radio transceivers 1762 and 1766 may include radios that arecompatible with any number of 3GPP (Third Generation PartnershipProject) specifications, notably Long Term Evolution (LTE), Long TermEvolution-Advanced (LTE-A), and Long Term Evolution-Advanced Pro (LTE-APro). It may be noted that radios compatible with any number of otherfixed, mobile, or satellite communication technologies and standards maybe selected. These may include, for example, any Cellular Wide Arearadio communication technology, which may include e.g. a 5th Generation(5G) communication systems, a Global System for Mobile Communications(GSM) radio communication technology, a General Packet Radio Service(GPRS) radio communication technology, or an Enhanced Data Rates for GSMEvolution (EDGE) radio communication technology, a UMTS (UniversalMobile Telecommunications System) communication technology, In additionto the standards listed above, any number of satellite uplinktechnologies may be used for the wireless network transceiver 1766,including, for example, radios compliant with standards issued by theITU (International Telecommunication Union), or the ETSI (EuropeanTelecommunications Standards Institute), among others. The examplesprovided herein are thus understood as being applicable to various othercommunication technologies, both existing and not yet formulated.

A network interface controller (NIC) 1768 may be included to provide awired communication to the cloud 1700 or to other devices, such as themesh devices 1764. The wired communication may provide an Ethernetconnection, or may be based on other types of networks, such asController Area Network (CAN), Local Interconnect Network (LIN),DeviceNet, ControlNet, Data Highway+, PROFIBUS, or PROFINET, among manyothers. An additional NIC 1768 may be included to allow connect to asecond network, for example, a NIC 1768 providing communications to thecloud over Ethernet, and a second NIC 1768 providing communications toother devices over another type of network.

The interconnect 1756 may couple the processor 1752 to an externalinterface 1770 that is used to connect external devices or subsystems.The external devices may include sensors 1772, such as accelerometers,level sensors, flow sensors, optical light sensors, camera sensors,temperature sensors, a global positioning system (GPS) sensors, pressuresensors, barometric pressure sensors, and the like. The externalinterface 1770 further may be used to connect the IoT device 1750 toactuators 1774, such as power switches, valve actuators, an audiblesound generator, a visual warning device, and the like.

In some optional examples, various input/output (I/O) devices may bepresent within, or connected to, the IoT device 1750. For example, adisplay or other output device 1784 may be included to show information,such as sensor readings or actuator position. An input device 1786, suchas a touch screen or keypad may be included to accept input. An outputdevice 1784 may include any number of forms of audio or visual display,including simple visual outputs such as binary status indicators (e.g.,LEDs) and multi-character visual outputs, or more complex outputs suchas display screens (e.g., LCD screens), with the output of characters,graphics, multimedia objects, and the like being generated or producedfrom the operation of the IoT device 1750.

A battery 1776 may power the IoT device 1750, although in examples inwhich the IoT device 1750 is mounted in a fixed location, it may have apower supply coupled to an electrical grid. The battery 1776 may be alithium ion battery, or a metal-air battery, such as a zinc-air battery,an aluminum-air battery, a lithium-air battery, and the like.

A battery monitor/charger 1778 may be included in the IoT device 1750 totrack the state of charge (SoCh) of the battery 1776. The batterymonitor/charger 1778 may be used to monitor other parameters of thebattery 1776 to provide failure predictions, such as the state of health(SoH) and the state of function (SoF) of the battery 1776. The batterymonitor/charger 1778 may include a battery monitoring integratedcircuit, such as an LTC4020 or an LTC2990 from Linear Technologies, anADT7488A from ON Semiconductor of Phoenix Ariz., or an IC from theUCD90xxx family from Texas Instruments of Dallas, Tex. The batterymonitor/charger 1778 may communicate the information on the battery 1776to the processor 1752 over the interconnect 1756. The batterymonitor/charger 1778 may also include an analog-to-digital (ADC)convertor that allows the processor 1752 to directly monitor the voltageof the battery 1776 or the current flow from the battery 1776. Thebattery parameters may be used to determine actions that the IoT device1750 may perform, such as transmission frequency, mesh networkoperation, sensing frequency, and the like.

A power block 1780, or other power supply coupled to a grid, may becoupled with the battery monitor/charger 1778 to charge the battery1776. In some examples, the power block 1780 may be replaced with awireless power receiver to obtain the power wirelessly, for example,through a loop antenna in the IoT device 1050. A wireless batterycharging circuit, such as an LTC4020 chip from Linear Technologies ofMilpitas, Calif., among others, may be included in the batterymonitor/charger 1778. The specific charging circuits chosen depend onthe size of the battery 1776, and thus, the current required. Thecharging may be performed using the Airfuel standard promulgated by theAirfuel Alliance, the Qi wireless charging standard promulgated by theWireless Power Consortium, or the Rezence charging standard, promulgatedby the Alliance for Wireless Power, among others.

The storage 1758 may include instructions 1782 in the form of software,firmware, or hardware commands to implement the techniques describedherein. Although such instructions 1782 are shown as code blocksincluded in the memory 1754 and the storage 1758, it may be understoodthat any of the code blocks may be replaced with hardwired circuits, forexample, built into an application specific integrated circuit (ASIC).

In an example, the instructions 1782 provided via the memory 1754, thestorage 1758, or the processor 1752 may be embodied as a non-transitory,machine readable medium 1760 including code to direct the processor 1752to perform electronic operations in the IoT device 1750. The processor1752 may access the non-transitory, machine readable medium 1760 overthe interconnect 1756. For instance, the non-transitory, machinereadable medium 1760 may be embodied by devices described for thestorage 1758 of FIG. 17 or may include specific storage units such asoptical disks, flash drives, or any number of other hardware devices.The non-transitory, machine readable medium 1760 may includeinstructions to direct the processor 1752 to perform a specific sequenceor flow of actions, for example, as described with respect to theflowchart(s) and block diagram(s) of operations and functionalitydepicted above.

In further examples, a machine-readable medium also includes anytangible medium that is capable of storing, encoding or carryinginstructions for execution by a machine and that cause the machine toperform any one or more of the methodologies of the present disclosureor that is capable of storing, encoding or carrying data structuresutilized by or associated with such instructions. A “machine-readablemedium” thus may include, but is not limited to, solid-state memories,and optical and magnetic media. Specific examples of machine-readablemedia include non-volatile memory, including but not limited to, by wayof example, semiconductor memory devices (e.g., electricallyprogrammable read-only memory (EPROM), electrically erasableprogrammable read-only memory (EEPROM)) and flash memory devices;magnetic disks such as internal hard disks and removable disks;magneto-optical disks; and CD-ROM and DVD-ROM disks. The instructionsembodied by a machine-readable medium may further be transmitted orreceived over a communications network using a transmission medium via anetwork interface device utilizing any one of a number of transferprotocols (e.g., HTTP).

It should be understood that the functional units or capabilitiesdescribed in this specification may have been referred to or labeled ascomponents or modules, in order to more particularly emphasize theirimplementation independence. Such components may be embodied by anynumber of software or hardware forms. For example, a component or modulemay be implemented as a hardware circuit comprising customvery-large-scale integration (VLSI) circuits or gate arrays,off-the-shelf semiconductors such as logic chips, transistors, or otherdiscrete components. A component or module may also be implemented inprogrammable hardware devices such as field programmable gate arrays,programmable array logic, programmable logic devices, or the like.Components or modules may also be implemented in software for executionby various types of processors. An identified component or module ofexecutable code may, for instance, comprise one or more physical orlogical blocks of computer instructions, which may, for instance, beorganized as an object, procedure, or function. Nevertheless, theexecutables of an identified component or module need not be physicallylocated together, but may comprise disparate instructions stored indifferent locations which, when joined logically together, comprise thecomponent or module and achieve the stated purpose for the component ormodule.

Indeed, a component or module of executable code may be a singleinstruction, or many instructions, and may even be distributed overseveral different code segments, among different programs, and acrossseveral memory devices or processing systems. In particular, someaspects of the described process (such as code rewriting and codeanalysis) may take place on a different processing system (e.g., in acomputer in a data center), than that in which the code is deployed(e.g., in a computer embedded in a sensor or robot). Similarly,operational data may be identified and illustrated herein withincomponents or modules, and may be embodied in any suitable form andorganized within any suitable type of data structure. The operationaldata may be collected as a single data set, or may be distributed overdifferent locations including over different storage devices, and mayexist, at least partially, merely as electronic signals on a system ornetwork. The components or modules may be passive or active, includingagents operable to perform desired functions.

Additional examples of the presently described method, system, anddevice embodiments include the following, non-limiting configurations.Each of the following non-limiting examples may stand on its own, or maybe combined in any permutation or combination with any one or more ofthe other examples provided below or throughout the present disclosure.

Example 1 is an edge control node of an industrial control systemcomprising: an IO subsystem for receiving data from a field device; anda system on a chip including: an ethernet switch communicatively coupledto a network; a microcontroller (MCU) to convert IO data from the IOsubsystem and send the data via the ethernet switch to an orchestrationserver via the network; and a central processing unit (CPU) initially inan inactive state to: receive an activation signal from theorchestration server via the ethernet switch; and change to an activatedstate in response to receiving the activation signal.

In Example 2, the subject matter of Example 1 includes, the activatedstate of the CPU including a low power mode and a high power mode.

In Example 3, the subject matter of Examples 1-2 includes, wherein theCPU is further to receive a deactivation signal from the orchestrationserver after a period of time in the activated state, and in response,return to the inactive state.

In Example 4, the subject matter of Examples 1-3 includes, wherein theedge control node is one of a plurality of edge control nodes in theindustrial control system, the plurality of edge control nodes includingat least one edge control node with an inactive CPU after the CPU isactivated.

In Example 5, the subject matter of Examples 1-4 includes, wherein theCPU is activated based on a determination by the orchestration serverthat the CPU is to be activated to satisfy a control strategy for theindustrial control system.

In Example 6, the subject matter of Examples 1-5 includes, wherein theethernet switch is a time-sensitive networking ethernet switch.

In Example 7, the subject matter of Examples 1-6 includes, wherein thenetwork has a ring topology with a bridge device connecting the networkto the orchestration server.

In Example 8, the subject matter of Examples 1-7 includes, wherein theactivation signal is received at the CPU directly from the MCU.

In Example 9, the subject matter of Examples 1-8 includes, wherein theCPU is further to receive processing instructions from the orchestrationserver, the CPU to execute the processing instructions when in theactivated state.

Example 10 is a method comprising: receiving IO data, using a processorof an orchestration server, via a bridge connecting the orchestrationserver to an edge control node, the IO data converted at amicrocontroller of the edge control node from data generated at an IOsubsystem to a packet sent by an ethernet switch; sending anauthenticated activation code to the edge control node to activate a CPUof the edge control node, the CPU initially in an inactivated state; andsending processing instructions to the CPU for execution.

In Example 11, the subject matter of Example 10 includes, determining,at the orchestration server, computational requirements of edge controlnodes in an industrial control system including the edge control node,wherein the CPU is activated based on a determination by theorchestration server that the CPU is to be activated to satisfy acontrol strategy for the industrial control system.

In Example 12, the subject matter of Examples 10-11 includes, receiving,at the orchestration server, an indication to activate a CPU of an edgecontrol node of the edge control nodes.

In Example 13, the subject matter of Examples 10-12 includes, whereinthe authenticated activation code is authenticated by the MCU before theCPU is activated.

In Example 14, the subject matter of Examples 10-13 includes, sending adeactivation code, from the orchestration server, to the CPU todeactivate the CPU.

In Example 15, the subject matter of Examples 10-14 includes, whereinthe edge control node is a node in a ring topology network with a bridgedevice connecting the network to the orchestration server.

Example 16 is an industrial control system comprising: a ring networkincluding a plurality of edge control nodes; an orchestration server; abridge connecting the orchestration server to the ring network; andwherein the plurality of edge control nodes includes, a first edgecontrol node comprising: a system on a chip including: a microcontroller(MCU) to convert IO data from an IO subsystem and send the data via anethernet switch to the orchestration server via the bridge; and acentral processing unit (CPU) in an initial inactive state to:

receive an activation signal from the orchestration server; and changeto an activated state in response to receiving the activation signal.

In Example 17, the subject matter of Example 16 includes, wherein theCPU is further to receive a deactivation signal from the orchestrationserver after a period of time in the activated state, and in response,return to the inactive state.

In Example 18, the subject matter of Examples 16-17 includes, whereinthe CPU is activated based on a determination by the orchestrationserver that the CPU is to be activated to satisfy a control strategy forthe industrial control system.

In Example 19, the subject matter of Examples 16-18 includes, whereinthe activation signal is received at the CPU directly from the MCU.

In Example 20, the subject matter of Examples 16-19 includes, whereinthe plurality of edge control nodes includes a second edge node with aCPU remaining in an inactive state after the CPU of the first edgecontrol node is activated.

In Example 21, the subject matter of Examples 16-20 includes, whereinthe orchestration server is further to send processing instructions tothe CPU for execution.

Example 22 is at least one machine-readable medium includinginstructions that, when executed by processing circuitry, cause theprocessing circuitry to perform operations to implement of any ofExamples 1-21.

Example 23 is at least one machine-readable medium includinginstructions, which when executed by a processor of an orchestrationserver, cause the processor to perform operations to: receiveinput/output (IO) data, the IO data received via a bridge connecting theorchestration server to an edge control node, wherein the IO data isconverted at a microcontroller (MCU) of the edge control node from datagenerated at an IO subsystem to a packet sent by a networking component;send an authenticated activation code to the edge control node toactivate a central processing unit (CPU) of the edge control node,wherein the CPU is initially placed in an inactivated state; and sendprocessing instructions to the CPU for execution.

In Example 24, the subject matter of Example 23 includes, wherein theoperations further cause the processor to determine computationalrequirements of edge control nodes in an industrial control systemincluding the edge control node, and wherein the CPU is activated basedon a determination by the orchestration server that activating the CPUsatisfies a control strategy for the industrial control system.

In Example 25, the subject matter of Examples 23-24 includes, whereinthe operations further cause the processor to receive an indication toactivate the CPU of the edge control node in the industrial controlsystem.

In Example 26, the subject matter of Examples 23-25 includes, whereinthe authenticated activation code is authenticated by the MCU before theCPU is activated.

In Example 27, the subject matter of Examples 23-26 includes, whereinthe operations further cause the processor to send a deactivation code,from the orchestration server, to the CPU to deactivate the CPU.

In Example 28, the subject matter of Examples 23-27 includes, whereinthe edge control node is a node in a ring topology network with a bridgedevice connecting the network to the orchestration server.

Example 29 is an apparatus comprising means to implement of any ofExamples 1-28.

Example 30 is a system to implement of any of Examples 1-28.

Example 31 is a method to implement of any of Examples 1-28.

Example 32 is at least one machine readable medium includinginstructions, which when executed by a computing system, cause thecomputing system to perform any of the operations of Examples 1-28.

Example 33 is an apparatus comprising respective means for performingany of operations of Examples 1-28.

Example 34 is a software defined industrial system, comprisingrespective devices and respective circuitry in the respective devices,with the respective circuitry configured to perform the operations ofany of operations of Examples 1-28.

Example 35 is an apparatus, comprising circuitry configured to performthe operations of any of the operations of Examples 1-28.

In Example 36, the subject matter of Example 35 includes, wherein theapparatus is a gateway enabling connection to adapted plurality of fielddevices, other device networks, or other network deployments.

In Example 37, the subject matter of Examples 35-36 includes, whereinthe apparatus is a device operably coupled to at least one sensor and atleast one actuator.

In Example 38, the subject matter of Examples 35-37 includes, whereinthe apparatus is an Edge Control Node device adapted for connection to aplurality of field devices.

In Example 39, the subject matter of Examples 35-38 includes, whereinthe apparatus is an Intelligent I/O Controller device adapted forconnection to a plurality of field devices.

In Example 40, the subject matter of Examples 35-39 includes, whereinthe apparatus is a Basic I/O Controller device adapted for connection toa plurality of field devices.

In Example 41, the subject matter of Examples 35-40 includes, whereinthe apparatus is a control server computing system adapted forconnection to a plurality of networked systems.

In Example 42, the subject matter of Examples 35-41 includes, whereinthe apparatus is a control processing node computing system adapted forconnection to a plurality of networked systems.

Example 43 is a networked system, comprising respective devicesconnected within a fog or cloud network topology, the respective devicescomprising circuitry configured to perform the operations of any ofExamples 1-28.

In Example 44, the subject matter of Example 43 includes, wherein therespective devices are connected via a real-time service bus.

In Example 45, the subject matter of Examples 43-44 includes, whereinthe network topology includes controller, storage, and computefunctionality for the software defined industrial system via a redundantpair of hosts.

In Example 4460, the subject matter of Examples 43-45 includes, whereinthe network topology includes controller, storage, and computefunctionalities for the software defined industrial system via separatephysical hosts.

Example 47 is a method for determining computational requirements ofedge control nodes in an industrial control system (e.g., a ringdeployment), such as at an orchestration server, receiving an indicationto activate CPUs of one or more edge control nodes, and sendingauthenticated activation codes to the edge control nodes with CPUs to beactivated.

Example 48 is a method for receiving an authenticated activation code atan edge control node, authenticating the code at the edge control node,and activating a CPU of the edge control node using a microprocessor(MCU) (e.g., a low performance processor).

In Example 49, the subject matter of Examples 47-48 includes: performingExamples 47-48 at a ring deployment of edge control nodes arranged by anorchestration system of an industrial control system.

In Example 50, the subject matter of Example 48 includes: receiving anupdate at the edge control node from the orchestration server todeactivate the CPU or place the CPU in a low power state.

30

What is claimed is:
 1. An edge control node of an industrial controlsystem comprising: an input/output (IO) subsystem for receiving a signalfrom a field device and generating IO data; and a system on a chipincluding: a networking component communicatively coupled to a network;a microcontroller (MCU) to convert the IO data from the IO subsystem andsend the converted data via the networking component to an orchestrationserver via the network; and a central processing unit (CPU) initially inan inactive state to change to an activated state in response to anactivation signal being received at the edge control node from theorchestration server via the networking component.
 2. The edge controlnode of claim 1, wherein the activated state of the CPU includes a lowpower mode and a high power mode.
 3. The edge control node of claim 1,wherein the CPU is further configured to receive a deactivation signalfrom the orchestration server after a period of time in the activatedstate, and in response, return to the inactive state.
 4. The edgecontrol node of claim 1, wherein the edge control node is one of aplurality of edge control nodes in the industrial control system, theplurality of edge control nodes including at least one edge control nodewith an inactive CPU after the CPU is activated.
 5. The edge controlnode of claim 1, wherein the CPU is activated based on a determinationby the orchestration server that the CPU is to be activated to satisfy acontrol strategy for the industrial control system.
 6. The edge controlnode of claim 1, wherein the networking component is a time-sensitivenetworking ethernet switch.
 7. The edge control node of claim 1, whereinthe network has a ring topology with a bridge device connecting thenetwork to the orchestration server.
 8. The edge control node of claim1, wherein the activation signal is received at the CPU directly fromthe MCU.
 9. The edge control node of claim 1, wherein the CPU is furtherto receive processing instructions from the orchestration server, theCPU to execute the processing instructions when in the activated state.10. At least one non-transitory machine-readable medium includinginstructions, which when executed by a processor of an orchestrationserver, cause the processor to perform operations to: receiveinput/output (IO) data, the IO data received via a bridge connecting theorchestration server to an edge control node, wherein the IO data isconverted at a microcontroller (MCU) of the edge control node from datagenerated at an IO subsystem to a packet sent by a networking component;send an authenticated activation code to the edge control node toactivate a central processing unit (CPU) of the edge control node,wherein the CPU is initially placed in an inactivated state; and sendprocessing instructions to the CPU for execution.
 11. The at least onemachine-readable medium of claim 10, wherein the operations furthercause the processor to determine computational requirements of edgecontrol nodes in an industrial control system including the edge controlnode, and wherein the CPU is activated based on a determination by theorchestration server that activating the CPU satisfies a controlstrategy for the industrial control system.
 12. The at least onemachine-readable medium of claim 10, wherein the operations furthercause the processor to receive an indication to activate the CPU of theedge control node in the industrial control system.
 13. The at least onemachine-readable medium of claim 10, wherein the authenticatedactivation code is authenticated by the MCU before the CPU is activated.14. The at least one machine-readable medium of claim 10, wherein theoperations further cause the processor to send a deactivation code, fromthe orchestration server, to the CPU to deactivate the CPU.
 15. The atleast one machine-readable medium of claim 10, wherein the edge controlnode is a node in a ring topology network with a bridge deviceconnecting the network to the orchestration server.
 16. An industrialcontrol system comprising: a ring network including a plurality of edgecontrol nodes; an orchestration server; a bridge connecting theorchestration server to the ring network; and wherein the plurality ofedge control nodes includes a first edge control node comprising: asystem on a chip including: a microcontroller (MCU) to convertinput/output (IO) data from an IO subsystem and send the converted datavia a networking component to the orchestration server via the bridge;and a processor in an initial inactive state to: receive an activationsignal from the orchestration server; and change to an activated statein response to receiving the activation signal.
 17. The industrialcontrol system of claim 16, wherein the processor is further configuredto receive a deactivation signal from the orchestration server after aperiod of time in the activated state, and in response, return to theinactive state.
 18. The industrial control system of claim 16, whereinthe processor is activated based on a determination by the orchestrationserver that activating the processor satisfies a control strategy forthe industrial control system.
 19. The industrial control system ofclaim 16, wherein the activation signal is received at the processordirectly from the MCU.
 20. The industrial control system of claim 16,wherein the plurality of edge control nodes includes a second edge nodewith a second processor remaining in an inactive state after theprocessor of the first edge control node is activated.
 21. Theindustrial control system of claim 16, wherein the orchestration serveris further configured to send processing instructions to the processorfor execution.
 22. The industrial control system of claim 16, whereinthe processor is a central processing unit (CPU).